Stempelator A Hybrid Stemmer for the Polish Language

Stemming algorithms, or in other words conflation of inflected forms of a word to a unique token, is a very popular tool in Information Retrieval and Data Mining. It often brings an improvement to the quality of results and decreases storage requirements of the processed information. Quite a few open source or free Polish stemmers emerged recently. They can be divided into two groups depending on the method used for conflating inflected terms: heuristic and dictionary-driven. In this work we provide a proposal that connects the two worlds into a hybrid stemmer. We show that a combination of two open source Polish stemmers—Lametyzator and Stempel—results in a stemmer that outperforms them in terms of stemming quality. We also provide an insightful analysis of experiments with different sizes and samples of data used to train Stempel, the heuristic part of Stempelator. We hope this analysis will be useful for further development of that stemmer as a stand-alone product. BibTEX entry: @techreport{ stempelator2005, author="Dawid Weiss", title="{Stempelator: A Hybrid Stemmer for the Polish Language}", institution="Institute of Computing Science, Pozna{\’n} University of Technology, Poland", type="Technical Report", number="RA-002/05", year="2005" } 04/02/05 00:24 CVS: Id: stempelator.tex,v 1.3 2005/02/03 21:38:10 dweiss Exp